Field of Art
The disclosure generally relates to the field of data management and specifically to storing and processing data organized as flexible records.
Description of the Related Art
Data stores typically consist of a system of tables that model application data in a structure of grouped fields known as a schema. Each of these fields has an assigned type and other constraints, such as character, byte-width and validity specifications. For example, a numeric field may only contain numbers up to a certain predetermined storage limit.
For data aggregation systems that collect data fields from several sources, such a rigid data storage infrastructure has several limitations. First, each source has its own set of fields which are described in a particular schema. Storing data from different sources in a rigid data store schema would thus require maintaining a separate schema for each source. Second, a particular data source may update its schema at any time, and the data store of the data aggregation system would need to be similarly altered requiring infrastructure changes and expensive data movement operations.
Accordingly, there is a need for a system that enables the storage of data fields received from different sources in a flexible manner.